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A  DoD  CCRP/NATO  Collaboration 


This  major  revision  to  the  Code  of  Best  Practice  for 
C2  Assessment  is  the  product  of  a  NATO  Research 
and  Technology  Organisation  (RTO)  sponsored 
Research  Group  (SAS-026).  It  represents  over  a 
decade  of  work  by  many  of  the  best  analysts  from 
the  NATO  countries.  A  symposium  (SAS-039)  was 
hosted  by  the  NATO  Consultation  Command  Control 
Agency  (NCSA)  and  provided  the  venue  for  a 
rigorous  peer  review  of  this  code. 

This  publication  is  the  latest  in  a  series  produced 
by  the  Command  and  Control  Research  Program 
(CCRP)  under  the  auspices  of  the  Assistant 
Secretary  of  Defense  (C3I).  The  CCRP  has 
demonstrated  the  importance  of  having  a  research 
program  focused  on  the  national  security 
implications  of  the  Information  Age.  The  research 
sponsored  and  encouraged  by  the  CCRP  contributes 
to  the  development  of  the  theoretical  foundations 
necessary  to  support  the  Information  Age 
transformation  of  the  Department.  Other  CCRP 
initiatives  are  designed  to  acquaint  military  and 
civilian  leaders  with  emerging  issues  related  to 
transformation.  This  CCRP  Publication  Series  is  a 
key  element  of  this  effort. 


Check  our  Web  site  for  the  latest  CCRP  activities  and  publications. 
This  code  is  also  avadable  on  our  Web  site. 


www.dodccrp.org 
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introduction 

Purpose 

NATO  has  produced  a  Code  of  Best  Practice 
(COBP)^  in  order  to  facilitate  high  quality 
assessments  supporting  decisionmaking  in  the  area  of 
Command  and  Control  (C2).  The  COBP  is  the  product 
of  international  collaboration  drawing  together  the 
operational  and  analytical  experience  of  leading  military 
and  civilian  defence  experts  from  across  the  NATO 
nations.  The  COBP  enhances  the  understanding  of  best 
practice  and  outlines  a  structured  process  for  the 
conduct  of  operational  assessment  for  C2,  which  is  the 
core  capability  of  Information  Age  defence  and  security. 

The  command  and  control  aspects  of  military  capability 
are  difficult  to  assess.  Use  of  the  COBP  will: 

•  Increase  the  likelihood  of  quality  products 
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•  Complete 

•  Relevant 

•  Transparent  (visible) 

•  Credible  (believable) 

•  Authoritative  (rigorous) 

•  Reduce  risk/cost  associated  with  producing  the  products 

The  COBP  emphasizes  the  critical  role  of 
decisionmakers  in  the  conduct  of  good  assessment 
and  describes  a  framework  to  help  to  structure  this 
participation.  The  COBP  should  be  used  by 
decisionmakers  to  frame  study  requirements,  provide 
additional  guidance,  and  review  products.  Use  of  the 
COBP  should  be  regarded  as  a  community  standard 
for  all  C2  assessments. 

This  decisionmaker’s  guide  aims  to  help 
decisionmakers  who  commission,  fund,  oversee,  and 
employ  C2  assessments.  It  provides  1 )  an  executive 
summary  of  the  NATO  Code  of  Best  Practice  (COBP) 
for  C2  Assessment,  2)  information  on  how 
decisionmakers  can  best  ensure  that  the  COBP  is 
adhered  to  by  those  carrying  out  C2  assessments, 
and  3)  guidance  on  the  limits  of  assessment  and  the 
use  of  C2  assessment  results. 

Background 

The  initial  version  of  the  NATO  COBP  for  C2 
Assessment  was  published  in  1 998,  as  the  culmination 
of  several  efforts  by  various  NATO  study  groups  to 
address  C2  assessment  challenges  and  practices. 
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While  this  initial  version  focused  on  Article  V  related 
operations,  this  version  of  the  COBP  addresses  all 
issues  of  C2  assessment  across  the  full  spectrum  of 
operations,  to  include  Operations  Other  Than  War 
(OOTW).  Experience  with  the  initial  version  of  the 
COBP  highlighted  the  need  to  produce  a  short, 
executive-level  version  of  the  COBP  aimed  specifically 
at  decisionmakers. 

Command  and  Control  Assessment 

C2^  is  recognized  as  a  critical  element  of  successful 
military  operations  and  a  key  aspect  of  Information 
Age  transformation.  Until  recently,  however,  physics- 
dominated  issues  of  military  operations,  rather  than 
C2  ones,  have  been  the  primary,  almost  exclusive, 
focus  of  military  analysis  and  assessments.  This, 
coupled  with  the  inherent  complexity  of  C2  (which 
involves  both  the  information  and  cognitive  domains), 
has  presented  the  assessment  community  with 
challenges  that  are  less  well  researched  and 
understood  and  with  a  tool  kit  that  is  clearly  lacking. 
The  NATO  COBP  for  C2  Assessment,  therefore,  has 
been  developed  to  help  C2  analysts  deal  with  these 
new  Information  Age  assessment  challenges  so  that 
they  can  improve  their  ability  to  take  on  analyses  of 
requirements,  analyses  of  alternatives,  research  on 
new  C2  concepts  and  capabilities,  and  support  real 
world  operations.  This  guide  addresses  how  the 
decisionmaker,  as  a  customer  of  C2  assessments, 
plays  a  key  role  in  such  assessments. 


4 


NATO  Code  of  Best  Practice  for  C2  Assessment 


Role  of  the  Decisionmaker 

The  decisionmaker  has  a  critical  role  in  any  assessment. 
The  decisionmaker  provides  the  initial  articulation  of  the 
problem  or  issue  at  hand  and  establishes  the  conditions 
under  which  the  effort  takes  place.  In  addition,  it  is  the 
decisionmaker  who  determines  how  the  assessment 
results  are  interpreted  and  whether  they  influence 
decisions.  When  the  decisionmaker  interposes 
subordinates  between  him  and  the  assessment  team 
he  must  understand  the  resulting  risks  and  take  action 
to  mitigate  them  where  possible. 

The  decisionmaker’s  role  in  a  C2  assessment  is  most 
definitely  a  hands-on  one.  Experience  shows  that  C2 
assessments  that  have  had  active  decisionmaker 
participation  are  more  likely  to  result  in  products  that 
satisfy  both  decisionmakers  and  the  members  of  the 
assessment  team.  Annex  A  provides  some  key 
questions  to  encourage  the  necessary  discussions 
between  the  decisionmaker  and  the  assessment  team 
at  various  points  in  the  study  effort.  This  guide  will 
provide,  at  a  high  level,  information  that  will  assist  the 
decisionmaker  in  assessing  the  answers.  The  full 
COBP  provides  a  more  detailed  treatment  of  these 
items  and  the  decisionmaker  may  want  to  consult  the 
full  COBP  as  appropriate. 

The  interaction  between  the  decisionmaker  and  the 
assessment  team  is  not  only  critical  to  getting  the  effort 
off  on  the  right  foot  but  is  essential  if  the  decisionmaker 
is  to  fully  understand  the  results  of  the  assessment 
and  the  assumptions  that  underlie  these  results. 
Furthermore  the  assessment  team  will,  at  various  times 
during  the  study,  have  important  choices  to  make. 
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Decisionmaker  involvement  in  these  choices  can  make 
the  difference  in  a  study’s  success  or  failure 

Establish  a  reasonable  set  of 
initial  conditions 

Stay  involved  throughout  the  study 
Maintain  consistent  focus  on  the 
real  problem 


Organization  of  the  Guide 

The  remainder  of  this  guide  is  organized  into  the 
following  four  sections: 

•  Preparing  for  Success 

•  Overview  of  the  Assessment  Process 

•  Monitoring  the  Assessment 

•  Reviewing  the  Products 

In  Preparing  for  Success,  the  things  that  the 
decisionmaker  can  do  to  enhance  the  likelihood  that 
the  effort  is  successful  are  discussed.  In  the  Overview 
of  the  Assessment  Process,  each  of  the  component 
steps  in  the  assessment  and  their  relationships  one  to 
another  are  discussed.  In  Monitoring  the  Assessment, 
key  considerations  are  identified,  any  one  of  which 
could  be  a  determining  factor  in  the  success  or  failure 
of  the  effort.  Reviewing  the  Products  addresses  the 
nature  of  the  products  a  C2  assessment  should  be 
expected  to  produce. 
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Preparing  for  Success 

The  are  two  major  prerequisites  for  successful  C2 
assessments.  First,  a  clear  and  unambiguous 
statement  of  the  decision  or  issue  to  be  addressed. 
Second,  a  well  qualified  assessment  team  with 
adequate  time  and  resources.  While  this  may  seem 
obvious,  all  too  often  either  one  or  both  of  these  two 
conditions  are  not  met.  In  these  cases,  it  may  be  best 
not  to  undertake  an  effort  that  will  be  doomed  to  failure 
from  the  start.  This  extreme  option  can  be  avoided  by 
properly  scoping  a  study  to  make  the  effort  feasible. 

To  begin  with,  every  effort  needs  to  be  made  to  ensure 
that  the  assessment  team  understands  the  problem 
or  issue  they  are  to  address.  Any  statement  of  the 
problem  will  contain  a  set  of  implicit  assumptions  and 
constraints  that  may  be  known  and  understood  by  the 
decisionmaker  but  not  by  members  of  the  assessment 
team.  An  effort  should  be  made  to  make  these  explicit. 

C2  assessments  are  often  undertaken  on  very 
“ambitious”  schedules  with  inadequate  resources. 
Experience  has  shown  that  C2  assessments  involve 
great  complexity,  difficulty  in  getting  appropriate  data, 
and  the  need  for  significant  sensitivity  analysis  to  deal 
with  uncertainty  and  risk.  Therefore,  plan 
conservatively.  This  will  avoid  the  situation  where  the 
effort  runs  out  of  time  and  resources  before  significant 
parts  of  the  assessment  can  be  completed  or  the 
assessment  team  is  forced  to  not  consider  or  measure 
or  analyze  key,  often  driving  aspects  of  the  problem. 

An  assessment  team  will  rarely  have  all  of  the  expertise 
or  experience  needed  to  do  the  job  right.  The  same  is 
true  for  empirical  data  and  the  results  of  other  efforts. 
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Access  to  the  right  people  and  information  is  every  bit 
as  important  as  having  a  well  articulated  problem  and 
adequate  time  and  resources.  It  is  essential  that  these 
needs  are  anticipated  before  the  study  begins  so  that 
appropriate  arrangements  for  access  are  arranged. 
Failure  to  accomplish  this  at  the  start  often  results  in 
delays,  cost  overruns,  and  compromises  the  quality 
of  the  products.  Additionally,  the  routine  collection  of 
data  during  exercises  and  military  operations  will 
facilitate  a  standing  collection  of  data  available  for 
future  C2  assessments. 

The  NATO  Code  of  Best  Practice  for  C2  Assessment 
has  proven  itself  to  be  useful  to  both  highly  experienced 
analysts  and  those  without  much  experience.  It  is 
recommended  that  decisionmakers  insist  that  the 
assessment  team  review  the  COBP  before  they 
develop  their  study  plans  and  inform  the  decisionmaker 
if,  when,  and  why  the  advice  and  or  processes 
contained  in  the  COBP  will  not  be  followed.  Any 
deviation  from  the  best  practices  described  in  the 
COBP  carry  risks  and  these  risks  need  to  be 
understood  by  decisionmakers  before  they  agree  to 
waive  portions  of  the  COBP  for  a  particular  effort. 

Ensure  that  the  problem  is 
understood 

Allocate  adequate  time  and 
resources  to  the  effort 
Plan  conservatively 
Ensure  study  leadership  has 
appropriate  breadth  and  experience 
Ensure  access  to  needed  expertise 
and  data 

Mandate  the  use  of  the  COBP 
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Overview  of  the  Assessment  Process 

Figure  1  depicts  the  C2  Assessment  Process.  This 
process  is  iterative.  It  is  applicable  to  any  type  of  C2 
assessment,  regardless  of  the  scope  or  focus  of  the 
assessment.  In  fact,  each  of  the  steps  in  the 
assessment  process  will  be  revisited  several  times 
during  the  course  of  the  effort. 


Figure  1 .  C2  Assessment  Process 


In  Problem  Formulation,  the  Assessment  Team 
answers  the  question  of  what  is  to  be  addressed  by 
the  effort.  The  Solution  Strategy  addresses  how  this 
will  be  accomplished.  The  remainder  of  the  process 
carries  out  the  solution  strategy  and  produces  the 
assessment  products.  Problem  Formulation  and 
development  of  the  Solution  Strategy  for  C2 
assessments  should  be  an  iterative  process. 
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The  decisionmaker  should  schedule  an  initial  review  with 
the  assessment  team  to  ensure  that  they  are  on  the  right 
track  after  the  assessment  team  has  gone  through  at 
least  one  full  iteration,  specifying  not  only  what  they  will 
do  but  also  thinking  about  each  of  the  remaining  steps  in 
the  process  in  some  detail.  Prior  to  this  review,  the 
assessment  team  should  have  their  study  plan  (this 
consists  of  both  a  statement  of  the  formulated  problem 
and  their  solution  approach)  peer  reviewed. 

It  should  be  expected  that  modifications  will  need  to 
be  made  in  either  the  solution  approach  or  in  the 
formulated  problem  as  greater  understanding  is 
gained.  The  decisionmaker  must  be  informed  when 
significant  changes  are  made.  Such  changes  may  for 
example  be  the  result  of  discovering  that  some  data 
needed  to  support  a  particular  measure  of  merit  (MoM) 
is  not  available.  In  such  cases  the  study  plan  needs  to 
be  revised  to  develop  a  surrogate  MoM,  collect  the 
appropriate  data,  or  conduct  a  sensitivity  analysis. 

Risks  and  uncertainties  are  an  inherent  part  of  C2 
assessments.  These  risks  and  uncertainties  can  not 
be  completely  eliminated,  therefore  they  must  be 
managed.  It  should  be  noted  that  each  iteration  ends 
with  a  consideration  of  residual  study  risk. 
Decisionmakers  should  expect  that  any 
recommendations  to  significantly  change  the  study 
plan  should  be  accompanied  by  an  explanation  of  the 
risks  to  study  objectives,  schedule,  and  costs.  Hence, 
the  continuous  involvement  of  the  decisionmaker 
ensures  that  the  study  effort  remains  appropriately 
focused  and  that  the  study  team  is  kept  aware  of 
decisions  and  developments  that  influence  the  study. 


10 


NATO  Code  of  Best  Practice  for  C2  Assessment 


Involvement  of  decisionmaker  is  key 
Process  must  be  iterative 
Initiai  review  needed  to  ensure 
proper  start 

Look  for  expiicit  treatment  of  risk 
and  uncertainty 

Expect  adoption  of  study  plan  as 
insight  is  gained 


Monitoring  the  Assessment 

The  first  two  steps  in  the  assessment  process  are 
Problem  Formulation  and  Solution  Strategy.  The 
product  of  these  two  steps  is  the  study  plan  which 
describes  the  what  and  how  of  the  assessment  effort. 

Participating  in  Probiem  Formuiation 

Effective  problem  formulation  is  fundamental  to  the 
success  of  any  assessment,  but  particularly  C2 
assessment  because  C2  issues  are  often  ill-defined 
and  complex.  The  problem  formulation  process 
identifies  the  context  of  the  study  and  aspects  of  the 
problem-related  issues  for  assessment.  The  context 
of  the  study  includes  the  geopolitical  environment,  aims 
and  objectives  of  the  study,  and  the  decisions  to  be 
supported  by  the  assessment.  Problem-related 
aspects  include  the  issues  to  be  addressed, 
assumptions,  high-level  MoMs,  independent  variables 
(both  controlled  and  uncontrolled),  and  constraints  on 
variables.  It  must  be  recognized  that  problem 
formulation  takes  time  and  must  precede  development 
of  the  assessment  concept  or  selection  of  the  solution 
approach  (including  methods  and  tools).  The 


Decisionmaker’s  Guide 


11 


decisionmaker  plays  a  critical  role  in  problem 
formulation  by  providing  an  understanding  of  the  real 
problem  to  be  addressed  and  how  study  results  will 
support  pending  decisions. 

The  team  should  be  expected  to  quickly  cover  the  whole 
problem  and  produce  an  initial  problem  formulation  to 
prevent  premature  narrowing  of  the  assessment  and 
to  allow  a  common  understanding  among  all  team 
members.  This  will  identify  the  key  issues  to  be 
addressed  and  define  the  context  of  the  study. 

As  a  result  of  this  process,  the  assessment  team  may 
discover  important  issues  related  to  the  problem  that 
need  to  be  discussed  with  the  decisionmaker. 

All  key  variables  should  be  included 
in  the  assessment 
Relationships  should  be  known  or 
hypothesized 

All  controllable  variables  should  be 
identified 

Key  assumptions  should  be  made 
explicit 


Monitoring  Soiution  Strategy 

A  solution  strategy  consists  of  the  specification  of  a  set 
of  sequential  and  parallel  analytical  steps,  often  involving 
several  methodologies  and  tools.  The  solution  strategy 
should  begin  with  what  is  known,  and  by  its  execution, 
lead  to  what  the  decisionmaker  desires  to  know — insight 
into  the  issue(s).  The  solution  strategy  can  be  simple, 
moderately  complicated,  or  extremely  complex. 
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The  development  of  a  solution  strategy  is  an  iterative 
process  that  should  strike  an  artful  balance  between 
what  the  team  would  like  to  do  and  what  is  possible  to 
do,  given  the  state  of  the  art,  available  data,  tools, 
schedule,  and  resources  available.  The  team  first 
elaborates  on  the  measures  that  are  to  be  evaluated 
in  the  study.  Using  these  measures  and  consideration 
of  human  and  organizational  factors,  a  conceptual 
model  of  the  assessment  is  developed.  The  conceptual 
model  is  based  on  the  issues  formulated  in  the  problem 
formulation  and  is  not  driven  by  the  availability  of  tools 
and  data.  The  conceptual  model  is  the  embodiment  of 
our  current  understandings  and  may  be  changed  by 
the  study  findings. 

Frequently  a  solution  strategy  becomes  complex, 
requiring  the  team  to  decompose  the  problem  into 
parts,  each  of  which  requires  assessment  with  its  own 
set  of  tools.  Taken  together,  the  solution  strategy 
consists  of  the  MoMs,  relevant  human  and 
organizational  factors,  specification  of  scenarios,  data 
collection  requirements,  and  methods  and  tools  to  be 
used  in  the  assessment. 

Solution  strategy  should  be  clearly 
articulated 

Potential  problems  should  be 
identified  with  contingency  plans 
Conceptual  model  should  not  be 
constrained  by  available  data  or  tools 
Conceptual  model  initially 
represents  current  understandings 
and  may  evolve 
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Approving  the  Study  Plan 

The  solution  strategy  is  documented  in  a  study  plan  that 
links  the  problem  formulation  and  solution  strategy 
together  in  one  plan.  The  study  plan  shoud  be  complete 
and  include:  MoMs,  Human  Organizational  Factors, 
Scenarios,  Methods  and  Tools,  Data  consideration.  Peer 
reviews,  in-process  reviews,  risk  and  uncertainty,  and 
products.  The  Study  Plan  should  be  developed  in  an 
iterative  fashion,  applying  guidance  and  feedback 
received  from  the  decisionmaker  and  other  stakeholders. 
It  is  important  that  the  Study  Plan  is  peer  reviewed. 

The  study  plan,  presented  by  the  assessment  team  to 
the  decisionmaker,  should  be  approved  before  significant 
resources  are  expended.  Often  the  study  plan  is 
supported  by  a  study  management  plan  to  guide, 
manage,  and  coordinate  the  effort.  The  study 
management  plan  may  have  subordinate  plans,  to 
include  an  analysis  plan,  modeling  and  simulation  plan, 
data  collection  plan,  configuration  management  plan, 
quality  assurance  plan,  review  plan,  deliverable  plan, 
security  plan,  as  well  as  a  study  risk  register,  and  glossary. 

Study  plan  should  be  complete 
Study  plan  should  be  realistic  in 
terms  of  schedule  and  resources 
and  allow  for  multiple  iterations 


Conducting  In-Process  Reviews 

The  in-process  reviews  are  critical  to  the  success  of 
the  effort.  They  should  be  at  key  junctures  including: 
After  initial  problem  formulation.  After  initial  version  of 
study  plan.  Prior  to  data  collection.  Between  collection 
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and  assessment,  Prior  to  draft  products.  These  are 
an  opportunity  for  the  decisionmaker  to  make  sure  that 
the  study,  if  completed  as  planned,  will  answer  the 
question  of  interest  and  also  to  make  appropriate 
changes  in  the  study  plan  to  correct  its  deficiencies. 
Furthermore,  reviews  help  the  decisionmaker  to 
harmonize  studies  conducted  in  parallel  under  their 
aegis  dealing  with  interrelated  issues. 

During  the  initial  review  the  decisionmaker  needs  to 
make  sure  that  all  of  the  key  assumptions  related  to 
the  assessment  are  fully  discussed  and  are 
appropriate.  In  addition,  these  reviews  should  include 
a  full  discussion  of  study  and  decision  risks. 

A  discussion  of  the  results  of  the  peer  review  and  a 
review  of  the  products  that  the  study  will  produce 
should  also  be  done  at  this  time. 

Conduct  reviews  at  key  junctures: 

After  initial  problem  formulation 

After  initial  version  of  study  plan 

Prior  to  data  collection 

Between  collection  and 

assessment 

Prior  to  draft  products 


Accepting  Measures  of  Merit  (MoMs) 

MoMs  are  central  to  meeting  the  objectives  of  all 
assessments.  Their  development  is  particularly 
challenging  for  C2  assessments,  given  the  nature  of 
the  problems  addressed.  It  is  recognized  that  no  single 
measure,  or  class  of  measures,  is  sufficient.  An 
orchestrated  set  of  MoMs  is  typically  required  for  C2 
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assessments.  The  COBP  has  adopted  the  following 
hierarchical  set  of  MoMs: 

•  Measures  of  Policy  Effectiveness  (MoPE)  that 
focus  on  policy  or  societal  outcomes 

•  Measures  of  Force  Effectiveness  (MoFE)  that 
focus  on  how  a  force  performs  its  mission  or  the 
degree  to  which  it  meets  its  objectives 

•  Measures  of  C2  Effectiveness  (MoCE)  that  focus 
on  the  impact  of  C2  systems  within  the 
operational  context 

•  Measures  of  Performance  (MoP)  that  focus  on  internal 
system  structure,  characteristics  and  behaviour 

•  Dimensional  Parameters  (DP)  that  focus  on  the 
properties  or  characteristics  inherent  in  the  C2  system 

A  good  C2  assessment  will  establish  relationships 
among  the  measures  used  in  the  assessment. 
Establishing  these  relationships  between  types  of 
measures,  however,  is  among  the  most  challenging 
parts  of  any  assessment.  It  is  important  to  recognize 
that  even  approximate  relationships  among  the  right 
MoMs  are  far  preferable  to  “precise”  relationships 
between  MoMs  that  do  not  adequately  reflect  the  key 
issues  at  hand.  Criteria  have  been  developed  that  help 
to  ensure  that  the  measures  that  are  selected  are  both 
valid  and  reliable.  For  a  measure  to  be  considered 
valid,  it  must  be  mission  oriented,  realistic,  appropriate, 
inclusive,  discriminatory,  meaningful,  simple,  relevant, 
and  generalisable.  For  a  measure  to  be  considered 
reliable,  it  must  be  discriminatory,  measurable/ 
observable,  quantitative,  objective,  sensitive,  and 
consistent.  A  proposed  measure  may  be  reliable  but 
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not  valid,  or  it  may  be  valid  but  not  reliable.  The 
decisionmaker  should  insist  that,  to  the  extent  possible, 
the  measures  to  be  used  are  valid,  reliable,  and,  if 
calculated,  will  provide  desired  information.  The  risk 
assessment  should  characterize  the  degree  to  which 
the  MoMs  may  not  be  valid  and  reliable  and  describe 
the  mitigating  actions  taken. 

Ensure  MoMs  are  valid,  measured 

reliably,  credible,  and  complete 


Considering  Human  and 
Organizationai  Factors 

The  human  dimension  largely  distinguishes  C2 
assessment  from  other  military  operations 
assessment.  C2  assessment  must  deal  with  distributed 
teams  including  military,  interagency,  coalition  and 
other  non-state  actors  operating  under  stress  and  their 
varying  decisionmaking  behaviours.  In  operations 
other  than  war,  particular  attention  must  be  paid  to 
behaviour  of  and  interaction  with  non-military 
organizations,  political  groups,  and  amorphous  groups 
such  as  crowds  and  refugees.  Thus,  the  formulation 
of  the  problem  and  the  development  of  solution 
strategies  cannot  be  completed  without  explicit 
consideration  of  both  human  and  organizational  issues. 

Human  performance  affects  behaviour  and  vice  versa. 
Human  performance  depends  on  psycho-physiological, 
and  cognitive  variables  such  as  stress,  fatigue,  sleep 
deprivation,  hunger,  alertness,  personality,  and 
predisposition,  as  well  as  on  ergonomic  and  external 
factors.  Individual  and  group  behaviors  are  also  the 
result  of  social  interaction  impacted  by  fear,  morale. 
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value  systems,  culture,  education,  and  religious 
backgrounds  of  individuals.  When  human  performance 
and/or  behavior  is  at  issue,  parameters  and/or  models 
will  need  to  reflect  those  issues.  Particularly  critical  is 
how  human  decisionmaking  is  addressed  in  the 
assessment,  and/or  in  models  used  in  the  assessment. 
The  human  factors  mentioned  above,  as  well  as  the 
command  style,  risk  style,  and  other  command  attributes 
that  impact  human  decisionmaking,  must  be  accounted 
for  in  the  modeling  and  assessment.  The  decisionmaker 
can  be  invaluable  in  this  regard  if  he/she  is  also  the 
subject  of  assessment  by  making  sure  the  assessment 
team  understands  their  approach  to  decisionmaking. 

There  is  a  strong  direct  link  between  human  and 
organizational  issues.  Properly  done,  organizational 
design  reflects  the  interaction  of  tasks  to  be  done,  the 
people  available  to  perform  them,  and  the  systems  or 
tools  that  support  them.  Because  of  the  need  for  co¬ 
evolution  organizational  design  is  often  a  key 
independent  variable.  Organizations,  then,  depend  on 
the  capabilities,  training,  and  experience  of  the  people 
in  the  C2  system.  Organizational  factors  must  be 
addressed  as  part  of  most  C2  analyses.  Organizational 
factors  include  structural  (e.g.  number  of  echelons,  span 
of  control),  functional  (e.g.  distribution  of  functions, 
information,  and  authority),  and  capacity  (e.g.  personnel, 
communications)  factors.  The  large  number  of 
organizational  variables  that  may  be  relevant  to  C2 
assessment  must  be  approached  carefully  and 
systematically.  When  possible,  organization  theory 
expertise  should  be  brought  into  the  team.  The 
assessment  team  will  be  challenged  to  identify  factors 
that  are  particularly  relevant  to  their  C2  assessment, 
and  to  identify  and  employ  appropriate  tools. 
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Explicit  consideration  of  human/ 
organization  performance 
Inclusion  of  appropriate  expertise 
on  the  team 


Approving  Scenarios 

The  selection  of  a  proper  set  of  scenarios  is  critical  to 
the  assessment.  Scenarios  consist  of  four  elements — 
a  context  (e.g.,  a  characterization  of  a  geopolitical 
situation),  the  participants  (e.g.,  intentions,  capabilities 
of  friendly,  hostile,  neutral),  the  environment  (e.g., 
natural — weather  and  man  made — mines),  and  the 
evolution  of  events  in  time.  Scenarios  can  be  approved 
or  unapproved.  Some  are  operational  scenarios, 
meaning  they  contain  additional  details  and  may  exist 
in  a  model.  Sometimes  smaller  scenarios,  called 
vignettes  are  developed  for  analytic  use. 

In  C2  assessments,  the  purpose  of  scenarios  is  to 
ensure  that  the  assessment  is  informed  by 
decisionmaker  planning  assumptions  and  the 
appropriate  range  of  opportunities  to  observe  the 
relevant  variables  and  their  interrelationships.  Although 
the  ideal  would  be  for  the  assessment  to  be  scenario 
independent,  rarely  does  this  happen  due  to  the 
breadth  and  complexity  of  C2.  Therefore,  scenarios 
must  be  considered  throughout  the  assessment 
process,  especially  during  problem  formulation.  In 
essence,  the  role  of  scenarios  is  to  define  a  set  of 
conditions  and  restrictions  to  enable  credible 
assessment  as  well  as  to  create  a  structure  within 
which  the  results  of  the  assessment  can  be  understood 
and  interpreted. 
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Scenarios  must  be  designed  or  selected  to  address 
C2  under  a  broad  range  of  circumstances.  C2 
assessments  need  to  use  multiple  scenarios,  no  single 
scenario  is  sufficient.  For  C2  assessment,  scenarios 
should  reflect  the  C2  organization,  processes,  and 
systems  relevant  to  the  assessment.  In  selecting 
scenarios  for  C2  assessment,  the  analyst  should 
ensure  that  the  scenarios  reflect  the  factors  that  have 
significant  impact  on  C2,  stress  C2  issues,  are  credible 
to  the  military,  are  credible  in  terms  of  civil-military 
objectives,  and  will  facilitate  the  study  design  process. 
Note  that  this  implies  a  broader  selection  of  scenarios 
than  is  normally  contained  within  current  sets  of 
“approved”  scenarios.  Due  to  their  critical  importance 
to  the  study,  the  decisionmaker  must  pay  particular 
attention  to  the  design  and  selection  of  scenarios. 

Ensure: 

Multiple  scenarios 

The  set  of  scenarios  cover  the 

range  of  relevant  situations 

Understanding  the  Methods  and  Tools 

There  is  a  broad  range  of  methods  and  tools  available 
to  be  applied  to  C2  assessment.  The  methods  and  tools 
fall  into  one  or  more  of  four  categories — data  collection/ 
generation  tools  (e.g.,  simulations,  exercises, 
experiments,  expert  elicitation,  real  world  operations), 
data  organization  tools  (e.g.,  causal  mapping,  multi¬ 
criteria  decision  analysis,  neural  nets),  “solving”  tools 
(e.g.,  mathematical  analysis,  linear  programming,  goal 
programming),  and  support  tools  (e.g.,  data  analysis, 
databases,  checklists,  spreadsheets).  Although  the 
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focus  of  recent  past  research  has  been  on  the 
development  of  computer  simulations,  virtually  any  of 
the  analytic  tools  in  the  analyst’s  inventory  could 
potentially  be  applied  to  C2  assessment. 

The  development  or  selection  of  tools  for  C2 
assessment  should  be  based  on  evaluation  of 
candidates  against  functionality-based  and 
performance-based  selection  criteria.  Functionality- 
based  selection  include  resolution/detail,  completeness/ 
scope,  the  functionality  coverage  provided,  the 
explicitness  of  entity  representation,  the  ability  to 
generate  appropriate  MoMs,  and  whether  the  tool  has 
been  verified,  validated,  and  accredited  (VV&A)  for  the 
intended  use.  Performance-based  selection  criteria 
include  responsiveness,  simplicity,  time  to  prepare/use, 
data  availability,  interoperability  with  other  tools, 
resources  required,  and  credibility  with  customers  and 
users.  Decisionmakers  should  refer  issues  regarding 
the  selection  of  methods  and  tools  to  peer  review. 

Although  the  assessment  team  will  employ  any  tool  that 
assists  in  addressing  study  issues,  credible  computer 
simulations  are  what  most  analysts  seek.  The 
development  of  C2  simulations,  especially  those  that 
link  C2  to  force  effectiveness,  has  been  the  subject  of 
much  research  recently.  Decisionmakers  should  be 
aware  that,  although  significant  progress  has  been 
made,  many  challenges  still  exist  in  the  modeling  of 
C2.  Among  these  challenges  are  representation  of 
human  behaviour,  linking/federating  models, 
representing  adversarial  entities,  dealing  with 
uncertainty  in  model  representations,  conduct  of 
sensitivity  analysis  with  models,  and  the  VV&A  process. 
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The  assessment  team  should  seek  approval  from  the 
decisionmaker  for  the  models  that  will  be  employed 
as  part  of  the  assessment.  Decisionmakers  should  ask 
how  C2  is  represented  in  proposed  models  and  how 
model  outputs  will  provide  the  data  necessary  to 
address  the  C2  issues  under  study. 

Study  needs  should  drive  tool 
selection,  not  vice  versa 
Seek  clear  rationale  for  tool  selection 
Multiple  tools  are  normally  needed 
Demand  explicit  linkage  from 
methods  and  tools  to  study  issues 


Comprehending  the  Data 

The  role  and  importance  of  data  in  C2  assessments  is 
underestimated  by  many.  The  ability  to  determine  what 
data  are  needed  and  the  ability  to  collect  these  data  in 
fact  determine  in  large  part  the  solution  strategy  (an 
example  of  the  importance  of  thinking  of  the  assessment 
process  as  iterative  in  nature).  There  may  be  many  types 
of  data  required,  to  include  scenario  data,  human 
performance  data,  and  systems  performance  data. 
Decisionmakers  and  analysts  prefer  data  to  be  sharp, 
certain,  complete,  and  consistent,  but  recognize  this 
will  not  always  be  the  case.  For  C2  assessment,  it  is 
particularly  difficult  to  meet  these  criteria. 

Hence,  not  all  C2  related  data  required/desired  by  the 
analysts  will  be  readily  available.  Some  of  it  will  have 
to  be  aggregated  or  derived  from  other  sources,  or 
perhaps  generated  from  original  sources.  C2  related 
data  can  be  obtained  from  a  variety  of  sources  to 
include  official  sources  (e.g.,  military  databases),  open 
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sources  (e.g.,  Internet),  legacy  studies,  and  subject 
matter  experts.  The  assessment  team  must 
understand  what  data  are  needed  and  in  what  form, 
who  owns  the  data  if  it  already  exists,  and  the  costs 
involved  in  buying,  collecting,  or  generating  required 
data.  These  costs  can  be  significant.  The 
decisionmaker  can  help  the  assessment  team  by 
assisting  in  identifying  available  or  potential  data 
sources.  Furthermore,  the  quality  of  the  study  results 
is  influenced  by  the  quality  of  data  used  as  much  as 
by  the  selection  of  respective  methods  and  tools. 

Articulate  the  assumptions  related 
to  collection 

Understand  data  anomalies  and 
adjustments  made  to  data 
Understand  data  sources 
Look  for  data  distributions,  not 
Just  averages 

Help  the  assessment  team  get  the 
data  they  need 


Understanding  Risks  and  Uncertainties 

There  are  risks  associated  with  the  decision  at  hand 
and  there  are  risks  related  to  the  assessment  process. 
Failure  to  deal  effectively  with  both  of  these  types  of 
risk  will  jeopardize  study  goals. 

Risk  is  commonly  defined  as  the  possibility  of  suffering 
harm  or  loss,  to  include  opportunity  loss.  Risk  often 
has  a  negative  connotation,  yet  “taking  risks”  can  also 
be  positive.  Uncertainty  can  be  defined  as  an  inability 
to  determine  a  variable  value  or  system  state  or  predict 
its  future  evolution.  Uncertainty  is  inherent  in  risk.  Risk 
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and  uncertainty  are  especially  prevalent  in  C2  and  in 
C2  assessment.  C2  issues  are  inherently  complex  and 
have  many  interacting  factors.  Additionally,  C2 
assessment  is  effected  by  uncertainties  in  scenarios, 
data,  and  models. 

Risk  can  be  dealt  with  in  C2  assessments  by  either 
reducing  the  uncertainty  that  underlies  the  risk,  by 
embracing  and  accounting  for  residual  uncertainties, 
or  by  communicating  the  risk  involved  and  adopting  a 
decision  strategy  that  mitigates  the  risks.  Uncertainty 
can  be  reduced  in  C2  assessment  in  a  number  of  ways, 
to  include  conducting  certification  of  data,  conducting 
proper  VV&A  of  models,  and  in  conducting  sensitivity 
analysis.  The  assessment  team  should  make  a  serious 
effort  to  illuminate  the  risks  and  to  conduct  an  explicit 
risk-based  assessment  (consideration  of  multiple 
potential  solutions).  With  problems  involving  human 
decisionmaking  the  analyst  must  be  aware  of  the 
diversity  of  courses  of  action  that  are  possible  as  a 
scenario  evolves.  The  analyst  should  also  attempt  not 
to  unduly  bound  the  problem  during  problem  formulation 
to  reduce  risk.  The  decisionmaker  can  help  the  analyst 
with  these  efforts  by  helping  to  identify  risks  and 
uncertainties  and  by  discussing  his  or  her  own  approach 
to  risk  and  uncertainty  regarding  the  study  issues  and 
the  decisions  to  be  made  based  on  the  assessment. 

Failure  to  adequately  deal  with  risk 
jeopardizes  study  goals 
Sensitivity  analyses  should  be 
provided  and  explained 
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Reviewing  the  Products 

Assessment  products  include  the  study  plan,  periodic 
status/progress  reviews,  and  the  final  report.  The 
decisionmaker  has  an  important  role  in  each.  The  study 
plan  should  be  presented  at  the  initial  review  and  the 
decisionmaker  should  approve  it.  The  study  plan 
should  serve  as  a  shared  understanding  between  the 
decisionmaker  and  among  the  study  participants. 
However  the  iterative  nature  of  good  C2  assessments 
means  the  study  plan  should  be  flexible. 

Periodic  reviews  should  be  chaired  by  the 
decisionmaker.  The  study  plan  should  go  through  a 
peer  review  process  prior  to  finalization.  In  addition  to 
these  products,  there  are  a  number  of  other  products 
that  best  practice  demands  be  produced  and  maintained 
during  the  course  of  the  study.  These  include  a  project 
journal,  a  study  design  and  implementation  plan,  a  data 
collection  plan,  a  data  analysis  plan,  and  a  risk  register. 
All  products  should  be  archiveable  and  readily 
accessible  by  the  community,  within  the  constraints  of 
security.  It  is  recommended  practice  that  a  version  be 
prepared  at  the  lowest  classification  possible  to  allow 
for  the  widest  distribution  possible  and  reuse  in  the 
assessment  community. 
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The  decisionmaker  should  be  able  to 
brief  the  study  and  explain  its  results 
Data  should  be  saved  in  a  form 
amenable  to  re-use 
Study  products  should  be  peer 
reviewed 

A  plan  should  be  developed  for 
archiving  and  dissemination 


^North  Atlantic  Treaty  Organisation  (NATO)  NATO  Code  of  Best 
Practice  for  C2  Assessment,  2002. 

^NATO  defines  02  as  “The  Organization,  Process,  Procedures, 
and  Systems  necessary  to  allow  timely  political  and  military 
decisionmaking  and  to  enable  military  commanders  to  direct 
and  control  military  forces.” 


ANNEX  A 


Decisionmaker’s 
Temporal 
Question  List 


This  annex  contains  key  questions  that  a 
decisionmaker  should  ask  the  C2  assessment 
team.  These  questions  are  organized  temporally 
according  to  the  following  phases  of  a  study  (prior  to 
the  study,  at  initial  review,  after  first  iteration,  and  at 
final  report). 

Prior  to  the  Study: 

•  Do  you  understand  what  decisions(s)  I  have  to 
make,  when  I  have  to  make  them,  and  the 
context  within  which  the  decision(s)  will  be 
made? 

•  Do  you  need  any  information  or  authorization 
from  me? 

•  Who  will  be  on  the  study  team? 

•  Are  there  adequate  skills,  experience  present 
in  the  team? 
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•  For  OOTW  studies,  in  particular,  are  there 
adequate  social  scientific  skills  in  the  team? 

•  Who  are  the  key  organizations/individuals  with 
whom  you  plan  to  interact  (e.g.,  stakeholders, 
data  providers,  review  team)? 

•  Have  you  coordinated  the  Terms  of 
Reference  with  them? 

•  How  do  you  plan  to  interact  with  them? 

•  Particularly  for  OOTW  studies,  how  will  you 
acquire  the  requisite  knowledge  of  the  culture/ 
historical  context? 

•  How  will  you  undertake  problem  formulation?  e.g., 

•  What  products  will  you  review/mine? 

•  What  methods  and  tools  are  applicable? 

•  When  will  key  events  occur?  (e.g.,  reviews, 
production  of  interim  products) 

At  the  Initial  Review: 

•  What  do  you  perceive  the  “real”  issues  to  be? 

•  What  assumptions  do  you  plan  to  make  to  scope 
the  effort? 

•  What  do  you  plan  to  use  for: 

•  High  level  MoMs? 

•  Scenarios  of  interest? 

•  Have  you  identified  any  additional  organizations/ 
individuals  with  whom  you  plan  to  coordinate? 
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•  How  do  you  plan  to  attack  the  problem? 

•  What  methodology  will  you  employ? 

•  How  will  you  treat  the  diverse  aspects  of  a 
mission  capability  package  (including  concept 
of  operations,  organization,  doctrine,  C2 
approach,  systems,  personnel,  facilities,  in 
other  words  everything  needed  to  field  a  real 
capability.) 

•  How  do  you  plan  to  address  organization/ 
human  issues? 

•  What  specific  methods  and  tools  will  you 
employ?  Why  do  you  think  they  are 
appropriate? 

•  What  data  will  you  employ?  Where  will  you 
get  them?  Why  do  you  think  they  are 
appropriate?  How  do  you  plan  to  make  the 
data  accessible  to  others?  How  do  you  plan 
to  depict  the  results  of  the  study? 

At  First  Iteration: 

•  What  specific  MoMs  were  selected?  What 
relationships  were  established  among  the 
MoMs? 

•  What  range  of  scenarios  were  selected?  Why? 

•  What  plans  do  you  have  to  illuminate  uncertainty/ 
sensitivity? 
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•  What  feedback  did  you  receive  from  the 
independent  review  team?  What  steps  did  you 
take  to  respond  to  it? 

•  What  do  you  plan  to  do  on  subsequent  iterations? 

•  Use  additional  tools? 

•  Consider  additional  scenarios,  assumptions? 

•  Modify  assessment  boundaries? 

At  Final  Report: 

•  What  are  the  major  findings,  recommendations? 

•  What  are  the  key  points  of  uncertainty/sensitivity? 

•  What  issues  were  not  addressed  that  should  be 
treated  in  subsequent  assessments? 

•  What  key  lessons  did  you  learn  with  respect  to 
methods,  tools,  and  data? 

•  What  steps  are  you  going  to  take  to  disseminate 
key  insights,  products  to  the  broader  community? 


Are  there  any  voids/issues  that  warrant 
further  research? 


